
<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

    <title>MultiNode LAVA &#8212; LAVA 2024.05 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="_static/doctools.js"></script>
    <script src="_static/sphinx_highlight.js"></script>
    <link rel="shortcut icon" href="_static/favicon.ico"/>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Writing MultiNode tests" href="writing-multinode.html" />
    <link rel="prev" title="Deploying test images using LXC" href="deploy-lxc.html" />
    <link rel="canonical" href="https://docs.lavasoftware.org/lava/multinode.html" />
  
<meta charset='utf-8'>
<meta http-equiv='X-UA-Compatible' content='IE=edge,chrome=1'>
<meta name='viewport' content='width=device-width, initial-scale=1.0, maximum-scale=1'>
<meta name="apple-mobile-web-app-capable" content="yes">
<script type="text/javascript" src="_static/js/jquery-1.12.4.min.js"></script>
<script type="text/javascript" src="_static/js/jquery-fix.js"></script>
<script type="text/javascript" src="_static/bootstrap-3.4.1/js/bootstrap.min.js"></script>
<script type="text/javascript" src="_static/bootstrap-sphinx.js"></script>


  </head><body>

  <div id="navbar" class="navbar navbar-default navbar-fixed-top">
    <div class="container">
      <div class="navbar-header">
        <!-- .btn-navbar is used as the toggle for collapsed navbar content -->
        <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
          <span class="icon-bar"></span>
        </button>
        <a class="navbar-brand" href="index.html"><span><img src="_static/lava.png"></span>
          LAVA</a>
        <span class="navbar-text navbar-version pull-left"><b>2024.05</b></span>
      </div>

        <div class="collapse navbar-collapse nav-collapse">
          <ul class="nav navbar-nav">
            
                <li><a href="genindex.html">Index</a></li>
                <li><a href="contents.html">Contents</a></li>
            
            
              <li class="dropdown globaltoc-container">
  <a role="button"
     id="dLabelGlobalToc"
     data-toggle="dropdown"
     data-target="#"
     href="index.html">Site <b class="caret"></b></a>
  <ul class="dropdown-menu globaltoc"
      role="menu"
      aria-labelledby="dLabelGlobalToc"><ul class="current">
<li class="toctree-l1"><a class="reference internal" href="index.html">Introduction to LAVA</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="contents.html">Contents</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="glossary.html">Glossary of terms</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="support.html">Getting support</a></li>
</ul>
</ul>
</li>
              
                <li class="dropdown">
  <a role="button"
     id="dLabelLocalToc"
     data-toggle="dropdown"
     data-target="#"
     href="#">Page <b class="caret"></b></a>
  <ul class="dropdown-menu localtoc"
      role="menu"
      aria-labelledby="dLabelLocalToc"><ul>
<li><a class="reference internal" href="#">MultiNode LAVA</a><ul>
<li><a class="reference internal" href="#using-lava-multinode-synchronization">Using LAVA MultiNode synchronization</a><ul>
<li><a class="reference internal" href="#multinode-results">MultiNode Results</a></li>
</ul>
</li>
<li><a class="reference internal" href="#lava-multinode-timeout-behavior">LAVA MultiNode timeout behavior</a><ul>
<li><a class="reference internal" href="#recommendations-on-timeouts-for-multinode">Recommendations on timeouts for MultiNode</a></li>
</ul>
</li>
<li><a class="reference internal" href="#running-a-server-on-the-device-under-test">Running a server on the device-under-test</a><ul>
<li><a class="reference internal" href="#making-use-of-third-party-servers">Making use of third party servers</a></li>
<li><a class="reference internal" href="#using-wrapper-scripts">Using wrapper scripts</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</ul>
</li>
              
            
            
              
                
  <li>
    <a href="deploy-lxc.html" title="Previous Chapter: Deploying test images using LXC"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">&laquo; Deploying tes...</span>
    </a>
  </li>
  <li>
    <a href="writing-multinode.html" title="Next Chapter: Writing MultiNode tests"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">Writing Multi... &raquo;</span>
    </a>
  </li>
              
            
            
            
            
              <li class="hidden-sm"></li>
            
          </ul>

          
            
<form class="navbar-form navbar-right" action="search.html" method="get">
 <div class="form-group">
  <input type="text" name="q" class="form-control" placeholder="Search" />
 </div>
  <input type="hidden" name="check_keywords" value="yes" />
  <input type="hidden" name="area" value="default" />
</form>
          
        </div>
    </div>
  </div>

<div class="container">
  <div class="row">
    <div class="body col-md-12 content" role="main">
      
  <section id="multinode-lava">
<span id="multinode"></span><span id="index-0"></span><h1>MultiNode LAVA<a class="headerlink" href="#multinode-lava" title="Permalink to this heading">¶</a></h1>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="actions-protocols.html#multinode-protocol"><span class="std std-ref">MultiNode Protocol</span></a> and
<a class="reference internal" href="writing-multinode.html#writing-multinode"><span class="std std-ref">Using the multinode protocol</span></a></p>
</div>
<p>LAVA multi-node support allows users to use LAVA to schedule, synchronize and
combine the results from tests that span multiple targets. Jobs can be arranged
as groups of devices (of any type) and devices within a group can operate
independently or use the MultiNode API to communicate with other devices in the
same group during tests.</p>
<p>Within a MultiNode group, devices of the same device type are assigned a role
and a <code class="docutils literal notranslate"><span class="pre">count</span></code> of devices to include into that role. Role labels must be
unique across the entire MultiNode job. Each role has a <code class="docutils literal notranslate"><span class="pre">device_type</span></code> and any
number of roles can have the same <code class="docutils literal notranslate"><span class="pre">device_type</span></code>. Each role can be assigned
device <code class="docutils literal notranslate"><span class="pre">tags</span></code>.</p>
<p>Once roles are defined, actions (including test images and test definitions)
can be marked as applying to specific roles (if no role is specified, all roles
use the action).</p>
<p>If insufficient boards exist to meet the combined requirements of all the roles
specified in the job, the job will be rejected.</p>
<p>If there are not enough idle boards of the relevant types to meet the combined
requirements of all the roles specified in the job, the job waits in the
Submitted queue until all devices can be allocated.</p>
<p>Each test job is put into a <a class="reference internal" href="glossary.html#term-target_group"><span class="xref std std-term">MultiNode group</span></a> and basic
information about other jobs in that group will be available in each test shell
using the MultiNode API.</p>
<section id="using-lava-multinode-synchronization">
<span id="multinode-synchronization"></span><span id="index-1"></span><h2>Using LAVA MultiNode synchronization<a class="headerlink" href="#using-lava-multinode-synchronization" title="Permalink to this heading">¶</a></h2>
<p>MultiNode is implemented using a <a class="reference internal" href="glossary.html#term-protocol"><span class="xref std std-term">protocol</span></a> which allows test writers to
access synchronization calls in any part of a test job by sending a request to
the <a class="reference internal" href="actions-protocols.html#multinode-protocol"><span class="std std-ref">MultiNode Protocol</span></a>.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="pipeline-writer-secondary.html#delayed-start-multinode"><span class="std std-ref">Delaying the start of a job using Multinode</span></a> which is used when
<a class="reference internal" href="pipeline-writer-secondary.html#writing-secondary-connection-jobs"><span class="std std-ref">Writing jobs using Secondary Connections</span></a></p>
</div>
<p>Once each board has booted the test image, the <a class="reference internal" href="multinodeapi.html#multinode-api"><span class="std std-ref">MultiNode API</span></a> will also be available for use within each test definition
using scripts placed into the default PATH by the LAVA overlay.</p>
<p>Unless two or more roles use the MultiNode API to synchronize operations at
some point within the test job submission, the test jobs will start at the same
time but run independently. Even if the test jobs in a MultiNode group are
identical, the time taken to download, deploy and boot into the test shell will
vary. There is no guarantee that a service will be available for another role
in the MultiNode group unless the test writer uses the synchronization
primitives in the MultiNode API. This also applies to tests where one role
needs to send data (like an IP address) to another role. One of the first tasks
for many MultiNode test jobs is to synchronize specific roles.</p>
<p>To synchronize all roles within the <a class="reference internal" href="glossary.html#term-target_group"><span class="xref std std-term">MultiNode group</span></a>, use
<a class="reference internal" href="multinodeapi.html#lava-sync"><span class="std std-ref">lava-sync</span></a>. If any role fails to execute this call, the entire group will
fail the synchronization.</p>
<p>In <strong>all</strong> roles in this MultiNode group:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-sync server</span>
</pre></div>
</div>
<p>To synchronize only specific roles, send a specific string using
<a class="reference internal" href="multinodeapi.html#lava-send"><span class="std std-ref">lava-send</span></a> and make the other role use <a class="reference internal" href="multinodeapi.html#lava-wait"><span class="std std-ref">lava-wait</span></a> with that same
string. Then send another message from the waiting role and make the first role
wait for the second message.</p>
<p>In the role acting as a server:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-send server</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-wait client</span>
</pre></div>
</div>
<p>In the role acting as a client:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-wait server</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-send client</span>
</pre></div>
</div>
<p>If one role is <strong>essential</strong> to all other roles in the test job, for example if
a role has to install and configure a server which is to be contacted by other
roles within the group, <a class="reference internal" href="actions-protocols.html#multinode-essential-roles"><span class="std std-ref">mark that role as essential</span></a>. When the job(s) marked with the essential role
fail, all test jobs in the MultiNode group will terminate.</p>
<p>To make your test job submissions more portable, it is recommended to use
<a class="reference internal" href="writing-tests.html#inline-test-definitions"><span class="std std-ref">inline test definitions</span></a> when calling the
MultiNode API from the test shell. All MultiNode API calls can also be executed
from <a class="reference internal" href="writing-tests.html#custom-scripts"><span class="std std-ref">custom scripts</span></a> although this can make things
harder to debug.</p>
<p>MultiNode synchronization calls will exit non-zero if the attempt times out or
fails in some other way. The test shell definition will then exit at this
point.</p>
<p>It is <strong>not</strong> recommended to wrap MultiNode synchronization calls in calls to
<code class="docutils literal notranslate"><span class="pre">lava-test-case</span></code> because if the API call fails, <code class="docutils literal notranslate"><span class="pre">lava-test-case</span></code> will
report a fail result but the test definition itself will continue as if the
synchronization succeeded. The synchronization calls themselves will create
results based on the operation requested.</p>
<section id="multinode-results">
<span id="index-2"></span><span id="id1"></span><h3>MultiNode Results<a class="headerlink" href="#multinode-results" title="Permalink to this heading">¶</a></h3>
<p>Each call to <a class="reference internal" href="multinodeapi.html#lava-send"><span class="std std-ref">lava-send</span></a>, <a class="reference internal" href="multinodeapi.html#lava-sync"><span class="std std-ref">lava-sync</span></a>, <a class="reference internal" href="multinodeapi.html#lava-wait"><span class="std std-ref">lava-wait</span></a> or
<a class="reference internal" href="multinodeapi.html#lava-wait-all"><span class="std std-ref">lava-wait-all</span></a> will generate a <a class="reference internal" href="glossary.html#term-test-case"><span class="xref std std-term">test case</span></a> with a <code class="docutils literal notranslate"><span class="pre">multinode-</span></code>
prefix in the current <a class="reference internal" href="glossary.html#term-test-suite"><span class="xref std std-term">test suite</span></a> of the results for this test job. If
the synchronization completes within the timeout, the result will be a
<code class="docutils literal notranslate"><span class="pre">pass</span></code>. If the attempt to synchronize times out, the result will be a
<code class="docutils literal notranslate"><span class="pre">fail</span></code>.</p>
<p>For example:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-wait server</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lava-send client</span>
</pre></div>
</div>
<p>Would generate test case results like <code class="docutils literal notranslate"><span class="pre">multinode-wait-server</span></code> and
<code class="docutils literal notranslate"><span class="pre">multinode-send-client</span></code>.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="writing-tests.html#test-definition-portability"><span class="std std-ref">Write portable test definitions</span></a></p>
</div>
</section>
</section>
<section id="lava-multinode-timeout-behavior">
<span id="index-3"></span><h2>LAVA MultiNode timeout behavior<a class="headerlink" href="#lava-multinode-timeout-behavior" title="Permalink to this heading">¶</a></h2>
<p>The submitted YAML includes a timeout value - in single node LAVA, this is
applied to each individual action executed on the device under test (not for
the entire job as a whole). i.e. the default timeout can be smaller than any
one individual timeout used in the YAML or internally within LAVA.</p>
<p>In MultiNode LAVA, this timeout is also applied to individual polling
operations, so an individual lava-sync or a lava-wait will fail on any node
which waits longer than the default timeout. The node will receive a failure
response.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="actions-protocols.html#multinode-essential-roles"><span class="std std-ref">Marking some roles as essential</span></a> - if your test job involves a
long running server and clients, marking the server as essential allows the
client test jobs to fail early instead of waiting for a long timeout.</p>
</div>
<section id="recommendations-on-timeouts-for-multinode">
<span id="multinode-timeouts"></span><h3>Recommendations on timeouts for MultiNode<a class="headerlink" href="#recommendations-on-timeouts-for-multinode" title="Permalink to this heading">¶</a></h3>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p><a class="reference internal" href="timeouts.html#timeouts"><span class="std std-ref">Timeouts</span></a></p>
</div>
<p>MultiNode operations have implications for the timeout values used in YAML
submissions. If one of the synchronization primitives times out, the sync will
fail and the job itself will then time out. One reason for a MultiNode job to
timeout is if one or more boards in the group failed to boot the test image
correctly. In this situation, all the other boards will continue until the
first synchronization call is made in the test definition for that board.</p>
<p>The time limit applied to a synchronization primitive starts when the board
makes the first request to the Coordinator for that sync. Slower boards may
well only get to that point in the test definition after faster devices
(especially KVM devices) have started their part of the sync and timed out
themselves.</p>
<p>Always review the protocol timeout and job timeouts in the YAML submission.
Excessive timeouts would prevent other jobs from using boards where the waiting
jobs have already failed due to a problem elsewhere in the group. If timeouts
are too short, jobs will fail unnecessarily.</p>
</section>
</section>
<section id="running-a-server-on-the-device-under-test">
<h2>Running a server on the device-under-test<a class="headerlink" href="#running-a-server-on-the-device-under-test" title="Permalink to this heading">¶</a></h2>
<p>If this server process runs as a daemon, the test definition will need to
define something for the device under test to actually do or it will simply get
to the end of the tests and reboot. For example, if the number of operations is
known, would be to batch up commands to the daemon, each batch being a test
case. If the server program can run without being daemonised, it would need to
be possible to close it down at the end of the test (normally this is the role
of the sysadmin in charge of the server box itself).</p>
<section id="making-use-of-third-party-servers">
<h3>Making use of third party servers<a class="headerlink" href="#making-use-of-third-party-servers" title="Permalink to this heading">¶</a></h3>
<p>A common part of a MultiNode setup is to download components from third party
servers but once the test starts, latency and connectivity issues could
interfere with the tests.</p>
</section>
<section id="using-wrapper-scripts">
<h3>Using wrapper scripts<a class="headerlink" href="#using-wrapper-scripts" title="Permalink to this heading">¶</a></h3>
<p>Wrapper scripts make it easier to test your definitions before submitting to
LAVA. The wrapper lives in a VCS repository which is specified as one of the
testdef_repos and will be available in the same directory structure as the
original repository. A wrapper script also helps the tests to fail early
instead of trying to do the rest of the tests.</p>
</section>
</section>
</section>


    </div>
      
  </div>
</div>
<footer class="footer">
  <div class="container">
    <p class="pull-right">
      <a href="#">Back to top</a>
      
    </p>
    <p>
        &copy; Copyright 2010-2019, Linaro Limited.<br/>
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br/>
    </p>
  </div>
</footer>
  </body>
</html>